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REMARKS 

In view of the above amendments and following remarks, favorable reconsideration in 
this application is respectfully requested. The Applicant further requests entry of the 
Amendment After Final dated October 4, 2010. 

Telephone Interview 

A telephone interview was conducted with the Examiner on November 11, 2010. The 
present Amendment incorporates changes based on that interview. Specifically, claims 44, 48 
and 50 have been amended to include further limitations which define over the prior art. If the 
Examiner has any questions or comments, it is respectfully requested that the Examiner contact 
the undersigned counsel. 

Claim Rejection - 35 USC 1 12 

The Examiner rejects claims 44, 48-49 under 35 USC 1 12, first paragraph, as failing to 
comply with the enablement requirement. The Examiner indicates that the specification fails to 
provide details on how the captured image, which is a bitmap, can be used to select a specific 
component of the object for displaying its annotating information. 

In response, the Applicant refers the Examiner to paragraphs [0022]-[0025] describes 

how the horizon is determined, with respect to Figs. 4-6. Paragraphs [0025]-[0028] outline how 
Page 7 of 16 

123593.00106/36053274v.l 



SerialNo.: 09/817,314 

Atty. Docket No. : 123593-00106 

Reply to Office Action of August 3, 2010 

points on an image are selected for annotation. Paragraph [0025] describes that for each feature 
in the viewing environment recorded in step (a), a matching feature is found in the image, or 
alternatively the extracted list can be globally matched to the pictorial data by finding the best 
over-all match of the viewer environment image. Paragraph [0026] discloses one particular 
example which employs the difference in color brightness to identify a specific part (in this case 
the horizon) of the displayed object; paragraph [0027] describes how a part of an image can be 
selected for annotation by means of clicking on it. 

With regards to claims 50 and 51, the Applicant refers to paragraphs [0037]-[0040] and 
[0045]- [0046] of the application. Paragraph [0040] describes that a radio beacon emits 
information about an object including the position of the object (in real space); the camera 
receives the information emitted by the beacon. Using the information it receives from the 
beacon, the camera establishes a correspondence between the object and its location in the 
image. Thus, real space positional information provided by the beacon, enables the camera to 
establish a correspondence between the object's real space position and its position in the image, 
so that the object in question can be identified in the image for annotation. As disclosed in 
paragraph [0045], centralized or decentralized processing can be employed for bringing objects, 
their representations, and their associated information into correspondence. 

Paragraph [0040] goes on to say that "A camera acquiring image and beacon signals can 
be aware of its position and shooting orientation so that, for placing the information [on the 
image], the camera can indicate the location of the beacon". Thus, the location of the beacon can 
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then be annotated on the image. Paragraph [0040] further discloses that alternatively, the camera 
and radio receiver can locate the beacon [on the image] through array processing. Paragraph 
[0037] illustrates a similar example wherein a radio provides information concerning the location 
(real space) of an object; the radio input is decoded in a module 1 1 . A camera input of an image 
representing a view and camera parameters such as location, direction and field of view, are 
obtained in a module 12. Data from modules 1 1 and 12 is used in module 13 to decide as to 
which objects are in view. In module 14, objects in view are annotated. 

Paragraphs [0045]- [0046] provide further detail as to how the camera establishes a 
correspondence between the object and its location in the image: " For bringing objects, their 
representations, and their associated information into correspondence, centralized or 
decentralized processing can be employed. For example, in cartographic annotation, all objects 
are maintained with their coordinates and associated information together in the same database. 
In decentralized processing this need not be the case, and hybrid arrangements also can be 
advantageous. For example, the names of restaurants in a locality can be obtained from a 
database such as the Yellow Pages, their regular menus from separate further databases, and their 
daily specials dispatched with a beacon. 

An object can be annotated based on the relative position of a display device with respect 
to the object, including their spatial relationship and the orientation of the device. In case of 
images or video, the relative position of an element on a screen can be calculated from camera 
parameters and relative position information of the camera with respect to the element. Where 
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annotation is displayed without displaying an image of the element, the information to be 
displayed can be selected based on location and orientation of a display device in relation to the 
element pointed to. Also, a beacon signal can be used that is sufficiently directional and/or 
having a sufficiently directional capture. The beacon signal can carry identifying information and 
possibly annotation information." 

Accordingly, it is respectfully submitted that the claimed invention is in compliance with 
35 USC 112, 1 st paragraph. 

Claim Rejection - 35 USC 103 

The Examiner rejects claims 44, 48-49 as unpatentable over Rose (Annotating Real- 
World Objects Using Augmented Reality) in view of Harrison (U.S. Patent No. 6,61 1,725); 
rejects claims 44, 48-49 and 51 over Rose and Ellenby (U.S. Patent No. 6,307,556; rejects claim 
50 over Rose and Sanderford (U.S. Patent No. 5,917,449). 

Rose only suggests selecting the objects to annotate by pointing at the actual object using 
a 6D pointing device illustrated on Fig. 3.1 and described in paragraphs 3.1 and 4.2. However, 
this requires a complicated and expensive pointer, such as a six degrees of freedom magnetic 
tracker able to sense the position and orientation of the pointer in space. Moreover, the solution 
disclosed by Rose is only applicable when the user has access to the physical object; it is not 
possible when this object is far away (such as a mountain in a landscape) or when the image or 
video was previously recorded and the physical object is not available anymore. 
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Thus, Rose does NOT teach selecting objects from the captured image, as required by at 
least claims 44, 48 and 49. Rather, Rose teaches the use of a physical pointing device to touch 
the physical object. 

Harrison discloses annotating for processing a design model generated by a computer 
aided design system. The method involves annotation of a drawing which has been created using 
CAD software (column 6 line 47- column 7 line 15). Thus, in considering the teaching of 
Harrison, the skilled person would understand that an element of a drawing created with CAD 
software can be selected for annotation. The skilled person would not learn how to select 
elements for annotation from a captured image . It is submitted that selecting an element from a 
captured image is very different from selecting elements from a CAD drawing, and that the 
method used to select elements from a CAD drawing could not be successfully used to select 
elements from an captured image, in particular since a captured image is formed of a bitmap 
whereas a CAD drawing is formed of characters. Thus, the skilled person could not arrive at the 
inventions of claims 44, 48 or 49 by combining the teaching of Rose and Harrison. 

Furthermore, it is submitted that a skilled person would not have attempted to combine 
the teaching of Rose and Harrison in the first place, as both documents teach completely 
different concepts which are incompatible with one another. Rose teaches selecting a component 
of a real 3D object, whereas Harrison teaches selecting a component from a 2D CAD drawing. It 
is not clear how the skilled person could combine the features disclosed in the two documents. 
For example, the 2D pointer used in Harrison could not be used in Rose, and neither could the 
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3D pointer of Rose be used in Harrison. The only reasonable combination of the teaching of 
these documents would lead the skilled person to prepare a 2D CAD drawing of the real 3D 
object and then to select a component for annotation from the 2D CAD drawing using a 2D 
pointer. However, this method would not involve the step of selecting an element from an image 
which has been captured by a camera, thus the combined teaching of Rose and Harrison cannot 
be said to obviate claim 44, 48, and 49. 

The Examiner further submits that claims 44, 48-48 and 51, are unpatentable over Rose in 
view of Ellenby. The applicant respectfully disagrees with the Examiner's view. The Examiner 
did not even try to show that mobile phones with camera were known at the filing date of the 
patent, or that the processing power available in mobile phones at that date was sufficient for 
processing images or computer vision. Therefore, the objection on obviousness is not supported. 
As the concept of including a camera in a mobile phone was not known at the filing date of the 
application, the feature of a mobile phone which includes a camera cannot be "simply a matter of 
design choice." 

In fact, it seems very unlikely that the one skilled in the art would even consider 
replacing the video camera of Rose. A mobile phone is of no use in the system of Rose, and the 
one skilled in the art has no reason to employ a mobile phone in the system of Rose, even if 
mobile phones with a video camera had been known at the filing date. It seems that the Rose 
system will only work with a fixed camera, due to the necessity to calibrate this camera with the 
pointer device and with the annotated object. The one skilled in the art will not consider a mobile 
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phone with camera for an application and system where a fixed video camera appears to be 
required. 

Rose describes an annotating system intended for annotating video images taken by a 
fixed video camera at a known location. A mobile phone with camera is poorly adapted to this 
purpose and would not be chosen by the skilled person. A mobile phone camera is only useful 
for annotating images taken on the go, such as landscape images or video of people in a meeting. 
Such an application of an annotating system has not, however, be considered or disclosed in any 
of the cited documents. 

The Examiner rejects claim 51 over Rose in view of Ellenby. The applicant respectfully 
disagrees with the Examiner's view. Claim 51 requires "using said position, said shooting 
direction and visual cues for relating a selected element to annotating data associated with the 
element." 

This feature requires, for example, a camera provided with a compass in order to 
determine its pointing direction. Rose only teaches identification based on a 6D pointer which is 
not part of the camera, and which is usually pointing at another direction than the camera, as 
clearly shown on Fig. 3.1 for example. Thus, Rose necessitates additional equipment whose 
position and orientation are determined, and does not deliver the same result. Moreover, Rose 
requires precise calibration between the video camera and the pointing device. In fact Rose even 
describes calibration as an essential component of augmented reality (paragraph 4); indeed, 
calibration is of utmost importance in order to map the position and direction of the pointer in the 
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physical world with the image taken by the video camera. Such a precise calibration seems only 
possible in a fixed setting, for example in order to annotate an engine placed at a known, fixed 
location, or using a calibration pattern as shown on Fig. 4.1 of Rose. 

Such a calibration between pointer and camera is not possible, or at least much more 
difficult, when the video camera is moving and when one does not know in advance its position 
and pointing direction. In this case, an external 6D pointing device is of no practical use since the 
annotating system cannot easily determine which element of the image is pointed. 

The invention disclosed in the present application concerns a method where the shooting 
direction of the camera itself is determined. It is therefore not needed to align or calibrate this 
direction with the image of the view. This results in a more simple method and system, and 
which completely solves the problems and errors of the calibration process. 

Ellenby does not use the shooting direction or visual cues for relating a selected element 
to annotating data associated with the element. Referring to column 7 lines 26-46, Ellenby 
discloses that relative elevation measurements depending on the location of the user are 
calculated. Ellenby also discloses calculating distance from an object and the direction to an 
object. However, Ellenby is completely silent as to how the relative elevation measurements, the 
distance measurements and direction are calculated. The relative elevation measurements, the 
distance measurements and direction are presumably determined based on the position of the 
camera. 
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With regards to the indicia 93, which identify the mountain and its features, it is only 
mentioned that the indicia 93 is "computer generated". Otherwise, Ellenby is completely silent as 
to how the computer generates indicia 93. In particular, Ellenby certainly does not mention that 
the shooting direction and visual cues are used to relate indicia 93 to the mountain in the image. 

Thus, it would appear that Ellenby uses the position of the camera only to annotate, the 
relative elevation of the mountain shown in the image, the distance from the mountain, and the 
direction to the mountain. However, the shooting direction and visual cues are not used in 
Ellenby for relating a selected element to annotating data associated with the element. 
Accordingly, even if the skilled person had considered the teaching of Rose in combination with 
Ellenby they would not arrive at the invention of claim 5 1 . 

The Examiner submits that claim 50 is unpatentable over Rose in view of Sanderford. 
The Applicant submits that the Examiner's objection is not well founded. Rose merely suggests 
a radio signal sent by the 6D pointer, but not a radio signal sent by a beacon or other radio- 
emitter attached to each object to annotate. Referring to column 10 lines 22-46 and column 12 
line 50- column 13 line 3, Sanderford discloses a radio beacon 406 which has an unknown 
location which can transmit absolute reference position information. The radio-beacon 406 of 
Sanderford could not be attached to each annotated part of the engine that Rose wants to 
annotate. Thus, the skilled person would not, and could not, combine the teaching of Rose and 
Sanderford to arrive at the invention of claim 50. 
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In addition, further to the telephone interview with the Examiner, claims 44, 48 and 50 
have been amended to require that the identification of the element is based on the location of the 
mobile phone and visual cues identified in the captured image. The prior art does not teach or 
suggest this feature of the invention. 

In the event there are any questions relating to this Amendment or to the application in 
general, it would be appreciated if the Examiner would telephone the undersigned attorney 
concerning such questions so that the prosecution of this application may be expedited. 

Please charge any shortage or credit any overpayment of fees to BLANK ROME LLP, 
Deposit Account No. 23-2185 (123593.00106). In the event that a petition for an extension of 
time is required to be submitted herewith and in the event that a separate petition does not 
accompany this response, Applicants hereby petition under 37 CFR 1 .136(a) for an extension of 
time for as many months as are required to render this submission timely. Any fee due is 
authorized above. 



600 New Hampshire Ave., N. W. 
Washington, D.C. 20037 
Telephone: (202) 944-3000 
Atty. Docket No.: 123593-00106 
Date: December 1, 2010 




By: /Peter S. Weissman/ 
Peter S. Weissman 
Reg. No. 40,220 
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